SENTENCE REPRESENTATIONS

Metric for Automatic Machine Translation Evaluation based on Universal Sentence Representations (N18-4)

Challenge:	State-of-the-art transformer models have achieved robust performance on a variety of NLP tasks.
Approach:	They propose to refine a pre-trained NLP model by detecting shuffled tokens . they use a sequential approach to train a model using random shuffling .
Outcome:	The proposed model achieves better performance on 4 out of 7 GLUE tasks.

Copied to clipboard

Hiroki Shimanaka, Tomoyuki Kajiwara, Mamoru Komachi

You Only Need Attention to Traverse Trees (P19-1)

Challenge:	Sentence representations can capture information that cannot be captured by local features based on character or word Ngrams.
Approach:	They propose a supervised regression model using universal sentence representations capable of capturing information that cannot be captured by local features based on character or word Ngrams.
Outcome:	The proposed model achieves state-of-the-art performance with only sentence representation features .

Copied to clipboard

Mahtab Ahmed, Muhammad Rifayat Samee, Robert E. Mercer

Language Models

Challenge:	Recent research has focused on sentence representations.
Approach:	They propose a tree-based model that captures phrase-level syntax and word-level dependencies by doing recursive traversal with attention.
Outcome:	a new model captures phrase-level syntax and word-level dependencies with attention.

Adaptive Reinforcement Tuning Language Models as Hard Data Generators for Sentence Representation (2024.lrec-main)

Copied to clipboard

Bo Xu, Yifei Wu, Shouang Wei, Ming Du, Hongya Wang

Learning Visually Grounded Sentence Representations (N18-1)

Challenge:	Existing methods use contrastive learning (CL) to learn effective sentence representations, but require extensive human annotation.
Approach:	They propose a reinforcement learning approach for fine-tuning small-parameter LLMs to generate high-quality hard contrastive data without human feedback.
Outcome:	The proposed method achieves state-of-the-art on seven semantic text similarity tasks.

Copied to clipboard

Douwe Kiela, Alexis Conneau, Allan Jabri, Maximilian Nickel

Multimodality

Challenge:	Unsupervised sentence representation models suffer from the grounding problem because of lack of association between symbols and external information.
Approach:	They train a sentence encoder to predict image features of a caption and use them as sentence representations.
Outcome:	The proposed model improves on word embeddings and word representations on standard benchmarks.

Multi-Domain Neural Machine Translation with Word-Level Domain Context Discrimination (D18-1)

Copied to clipboard

Jiali Zeng, Jinsong Su, Huating Wen, Yang Liu, Jun Xie, Yongjing Yin, Jianqiang Zhao

Text Generation

Challenge:	Experimental results on Chinese-English and English-French multi-domain translation tasks demonstrate the effectiveness of the proposed model.
Approach:	They propose to use mixed-domain parallel sentences to construct a unified model that allows translation to switch between different domains.
Outcome:	The proposed model distinguishes and exploits word-level domain contexts on Chinese-English and English-French translation tasks.

Hyper-CL: Conditioning Sentence Representations with Hypernetworks (2024.acl-long)

Copied to clipboard

Young Yoo, Jii Cha, Changhyeon Kim, Taeuk Kim

Modeling Inter-Aspect Dependencies for Aspect-Based Sentiment Analysis (N18-2)

Challenge:	Existing approaches to sentence embeddings do not capture fine-grained semantics of sentences.
Approach:	They propose a method that integrates hypernetworks with contrastive learning to generate conditioned sentence representations.
Outcome:	The proposed method narrows the performance gap with the bi-encoder architecture while maintaining the time efficiency characteristic of the tri-encoding approach.

Copied to clipboard

Devamanyu Hazarika, Soujanya Poria, Prateek Vij, Gangeshwar Krishnamurthy, Erik Cambria, Roger Zimmermann

Does Transliteration Help Multilingual Language Modeling? (2023.findings-eacl)

Challenge:	Present neural-based models exploit aspect and its contextual information in the sentence but ignore inter-aspect dependencies.
Approach:	They propose to combine aspect-based sentiment analysis with temporal dependency processing to incorporate this pattern into a sentence.
Outcome:	The proposed approach is based on the SemEval 2014 dataset and shows that it is effective for predicting sentiments of aspects in sentences with multiple aspects.

Copied to clipboard

Ibraheem Muhammad Moosa, Mahmud Elahi Akhter, Ashfia Binte Habib

Efficient Information Extraction in Few-Shot Relation Classification through Contrastive Representation Learning (2024.naacl-short)

Challenge:	Script diversity presents a challenge to Multilingual Language Models by reducing lexical overlap . Script diversification can be used to improve performance of MLLMs by transliterating closely related languages to a common script.
Approach:	They empirically measure the effect of transliteration on MLLMs by focusing on Indic languages . they find that transliterations benefit low-resource languages without negatively affecting high-resourced ones .
Outcome:	The proposed transliteration-based model learns sentences that are more similar across languages.

Copied to clipboard

Philipp Borchert, Jochen De Weerdt, Marie-Francine Moens

Responsible & Trustworthy NLP

Challenge:	Relation classification (RC) models extract rich information from sentences with limited labeled instances.
Approach:	They propose to combine multiple sentence representations with contrastive learning to enhance information extraction by combining multiple sentence and entity tokens.
Outcome:	The proposed approach is able to extract discriminative information from multiple representations and contrastive learning.

Parameter-free Sentence Embedding via Orthogonal Basis (D19-1)

Copied to clipboard

Ziyi Yang, Chenguang Zhu, Weizhu Chen

Evaluation Benchmarks and Learning Criteria for Discourse-Aware Sentence Representations (D19-1)

Challenge:	Existing methods to build sentence embeddings are parameterized and require training to optimize their parameters.
Approach:	They propose a non-parameterized method to combine pre-trained word embeddings into sentence representations using an orthogonal basis of the word vector subspace and its surrounding context.
Outcome:	The proposed method shows superior performance on 11 downstream NLP tasks and is competitive to other methods relying on large amounts of labelled data or prolonged training time.

Copied to clipboard

Mingda Chen, Zewei Chu, Kevin Gimpel

Incorporating Visual Semantics into Sentence Representations within a Grounded Space (D19-1)

Challenge:	Prior work on pretrained sentence embeddings and benchmarks focused on the capabilities of stand-alone sentences.
Approach:	They propose a test suite of tasks to evaluate whether sentence representations include broader context information.
Outcome:	The proposed training objectives help to encode different aspects of information in document structures.

Copied to clipboard

Patrick Bordes, Eloi Zablocki, Laure Soulier, Benjamin Piwowarski, Patrick Gallinari

Visual Data in NLP

Challenge:	Language grounding is an active field aiming at enriching textual representations with visual information.
Approach:	They propose to transfer visual information to textual representations by learning an intermediate representation space: the grounded space.
Outcome:	The proposed model outperforms the previous state-of-the-art on classification and semantic relatedness tasks.

Matching Varying-Length Texts via Topic-Informed and Decoupled Sentence Embeddings (2024.findings-naacl)

Copied to clipboard

Xixi Zhou, Chunbin Gu, Xin Jie, Jiajun Bu, Haishuai Wang

On the Evaluation of Semantic Phenomena in Neural Machine Translation Using Natural Language Inference (N18-2)

Challenge:	Existing approaches to matching text with non-comparable lengths are limited due to truncation issues.
Approach:	They propose a model that decouples sentences and embeds them into natural sentences for matching texts of significantly different lengths.
Outcome:	The proposed model matches texts of significantly different lengths across three well-studied datasets.

Copied to clipboard

Adam Poliak, Yonatan Belinkov, James Glass, Benjamin Van Durme

Machine Translation

Challenge:	Existing methods to investigate whether sentence representations from NMT systems capture distinct semantic phenomena are limited.
Approach:	They propose a process to investigate the extent to which sentence representations arising from neural machine translation systems encode distinct semantic phenomena.
Outcome:	The proposed model is suited to supporting inferences at the syntax-semantics interface, compared to anaphora resolution requiring world knowledge.

A Frame-based Sentence Representation for Machine Reading Comprehension (2020.acl-main)

Copied to clipboard

Shaoru Guo, Ru Li, Hongye Tan, Xiaoli Li, Yong Guan, Hongyan Zhao, Yueping Zhang

Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoning (2021.acl-long)

Challenge:	Existing machine learning approaches do not have above semantic knowledge to address complicated MRC questions.
Approach:	They propose a frame-based Sentence Representation method which integrates frame semantic knowledge to facilitate sentence modelling.
Outcome:	The proposed method performs better than state-of-the-art methods on machine reading comprehension task.

Copied to clipboard

Bill Yuchen Lin, Seyeon Lee, Xiaoyang Qiao, Xiang Ren

Reasoning

Challenge:	Using multilingual language models, commonsense reasoning research has been limited to English.
Approach:	They propose a Mickey Probe task to evaluate commonsense across languages . they propose X-CSQA and XCODAH datasets to be translated to 14 languages based on the Mickey corpus .
Outcome:	The proposed method significantly improves sentence representations beyond English.

Is Anisotropy Truly Harmful? A Case Study on Text Clustering (2023.acl-short)

Copied to clipboard

Mira Ait-Saada, Mohamed Nadif

Text Clustering

Challenge:	Contextualized pre-trained representations are widely used as input to various tasks such as information retrieval, anomaly detection and document clustering.
Approach:	They propose to examine the impact of different transformations on isotropy and performance to assess the true impact of anisotropi.
Outcome:	The proposed model is based on a clustering task and shows that it has limited impact on expressiveness and closeness.

A Comprehensive Survey of Sentence Representations: From the BERT Epoch to the CHATGPT Era and Beyond (2024.eacl-long)

Copied to clipboard

Abhinav Ramesh Kashyap, Thanh-Tung Nguyen, Viktor Schlegel, Stefan Winkler, See-Kiong Ng, Soujanya Poria

Hi-Transformer: Hierarchical Interactive Transformer for Efficient and Effective Long Document Modeling (2021.acl-short)

Challenge:	Sentence representations are a critical component in NLP applications such as retrieval, question answering, and text classification.
Approach:	They present a systematic review of the literature on sentence representations focusing mostly on deep learning models.
Outcome:	The proposed methods highlight the key contributions and challenges in this area and suggest potential avenues for improving the quality and efficiency of sentence representations.

Copied to clipboard

Chuhan Wu, Fangzhao Wu, Tao Qi, Yongfeng Huang

Are BLEU and Meaning Representation in Opposition? (P18-1)

Challenge:	Existing approaches to model long documents are difficult due to the quadratic complexity of text length.
Approach:	They propose a hierarchical interactive Transformer for efficient long document modeling.
Outcome:	Extensive experiments on three benchmark datasets validate the efficiency and effectiveness of Hi-Transformer in long document modeling.

Copied to clipboard

Ondřej Cífka, Ondřej Bojar

Sentence Bottleneck Autoencoders from Transformer Language Models (2021.emnlp-main)

Challenge:	Empirical evaluation suggests that the better the translation quality, the worse the learned sentence representations serve in a wide range of classification and similarity tasks.
Approach:	They propose several variations of the attentive NMT architecture to bring this meeting point back . they propose to use a structured fixed-size representation of the input to produce static representations of input sentences.
Outcome:	The proposed architecture improves translation quality and performance in a range of tasks.

Copied to clipboard

Ivan Montero, Nikolaos Pappas, Noah A. Smith

How Transliterations Improve Crosslingual Alignment (2025.coling-main)

Challenge:	Existing methods for pretraining a language model on text have been used for building models in NLP, but they do not work for sentence representations derived from pretrainer models based on tokens or basic pooling operations.
Approach:	They propose to build a sentence-level autoencoder from a pretrained transformer language model.
Outcome:	The proposed model achieves better quality than previous methods on text similarity and style transfer tasks while using fewer parameters than large pretrained models.

Copied to clipboard

Yihong Liu, Mingyang Wang, Amir Hossein Kargaran, Ayyoob ImaniGooghari, Orgest Xhelili, Haotian Ye, Chunlan Ma, François Yvon, Hinrich Schütze

Multilinguality

Challenge:	Recent studies show that post-aligning multilingual pretrained language models improve crosslingual alignment, but it is unclear how and why this is achieved.
Approach:	They propose to explicitly evaluate crosslingual alignment by adding transliterations to models using original and transliterated data.
Outcome:	The proposed approach improves crosslingual alignment even for random sentences.

Multi-label Categorization of Accounts of Sexism using a Neural Framework (D19-1)

Copied to clipboard

Pulkit Parikh, Harika Abburi, Pinkesh Badjatiya, Radhika Krishnan, Niyati Chhaya, Manish Gupta, Vasudeva Varma

Text Classification

Challenge:	Sexism manifests in blatant as well as subtle ways, authors say . existing work on sexism classification has limitations in terms of categories used . authors: categorization of accounts of sexist behavior can aid in countering sextism .
Approach:	They propose a neural solution that can combine sentence representations with distributional and linguistic word embeddings.
Outcome:	a new method outperforms deep learning and traditional methods by an appreciable margin . the proposed method outpersforms several deep learning as well as traditional baselines by an approval margin compared to baselines .

Are the Best Multilingual Document Embeddings simply Based on Sentence Embeddings? (2023.findings-eacl)

Copied to clipboard

Sonal Sannigrahi, Josef van Genabith, Cristina España-Bonet

Hierarchical User and Item Representation with Three-Tier Attention for Recommendation (N19-1)

Challenge:	obtaining document embeddings at document level is challenging due to computational requirements and lack of appropriate data.
Approach:	They compare methods to produce document-level representations from sentences based on LASER, LaBSE, and Sentence BERT pre-trained multilingual models.
Outcome:	The proposed methods produce document-level representations from sentences in 8 languages . the results show that a clever combination of sentence embeddings is usually better than encoding the full document as a single unit.

Copied to clipboard

Chuhan Wu, Fangzhao Wu, Junxin Liu, Yongfeng Huang

Movie Plot Analysis via Turning Point Identification (D19-1)

Challenge:	Existing methods to learn user and item representations from reviews are limited . existing methods learn user representations based on ratings given by users .
Approach:	They propose a hierarchical user and item representation model with three-tier attention to learn user and items from reviews for recommendation.
Outcome:	The proposed model can learn user and item representations from reviews on four benchmark datasets.

Copied to clipboard

Pinelopi Papalampidi, Frank Keller, Mirella Lapata

Challenge:	Using computational literary analysis, we analyze novels, plays, and screenplays for their turning points.
Approach:	They propose to use turning points to analyze screenplays and plot synopses as tools for analysis . they propose to build a neural network model that identifies turning points in plot synoopse .
Outcome:	The proposed model outperforms baselines based on state-of-the-art sentence representations and expected position of turning points.

Neural Deepfake Detection with Factual Structure of Text (2020.emnlp-main)

Copied to clipboard

Wanjun Zhong, Duyu Tang, Zenan Xu, Ruize Wang, Nan Duan, Ming Zhou, Jiahai Wang, Jian Yin

Structured Data in NLP

Challenge:	Existing approaches to deepfake detection typically represent documents with coarse-grained representations, but they struggle to capture factual structures of documents.
Approach:	They propose a graph-based model that captures factual structures of documents for deepfake detection.
Outcome:	The proposed model improves strong base models built with RoBERTa on two public deepfake datasets.

Effective and Efficient Query-aware Snippet Extraction for Web Search (2022.emnlp-main)

Copied to clipboard

Jingwei Yi, Fangzhao Wu, Chuhan Wu, Xiaolong Huang, Binxing Jiao, Guangzhong Sun, Xing Xie

What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties (P18-1)

Challenge:	Existing methods to extract webpage snippets ignore contextual information of webpages, which may be sub-optimal.
Approach:	They propose a query-aware webpage snippet extraction method called DeepQSE that captures contextual information of webpages.
Outcome:	The proposed method can significantly improve the performance of DeepQSE without affecting its performance.

Copied to clipboard

Alexis Conneau, German Kruszewski, Guillaume Lample, Loïc Barrault, Marco Baroni

Content Selection in Deep Learning Models of Summarization (D18-1)

Challenge:	a lack of understanding of the properties of sentence embeddings is limiting the use of the techniques.
Approach:	They propose 10 probing tasks designed to capture simple linguistic features of sentences . they use three different encoders to train embeddings in eight different ways .
Outcome:	The proposed tasks capture key linguistic features of sentences, but they are difficult to infer from them.

Copied to clipboard

Chris Kedzie, Kathleen McKeown, Hal Daumé III

Summarization

Challenge:	Using deep learning models, we find that word embedding does not improve performance over simpler models.
Approach:	They propose to use sentence embedding to perform content selection across multiple domains . they propose to propose two alternative models that use auto-regressive sentence extraction .
Outcome:	The proposed models improve performance across news, personal stories, meetings, and medical articles.

An Evaluation Dataset for Identifying Communicative Functions of Sentences in English Scholarly Papers (2020.lrec-1)

Copied to clipboard

Kenichi Iwatsuki, Florian Boudin, Akiko Aizawa

Improved Universal Sentence Embeddings with Prompt-based Contrastive Learning and Energy-based Learning (2022.findings-emnlp)

Challenge:	Formulaic expressions are used by authors of scientific papers because they convey specific communicative functions in the rhetorical structure of papers.
Approach:	They created a manually annotated dataset to detect formulaic expressions in sentences using a seed list of labelled formulaic words.
Outcome:	The proposed dataset can detect communicative functions in sentences using a seed list of labelled expressions from scholarly papers in the ACL Anthology.

Copied to clipboard

Yuxin Jiang, Linhan Zhang, Wei Wang

Discourse Self-Attention for Discourse Element Identification in Argumentative Student Essays (2020.emnlp-main)

Challenge:	Existing contrastive methods for learning universal sentence embeddings have limitations due to their over-parameterization and poor performance under domain shift settings.
Approach:	They propose to integrate an Energy-based Hinge loss to enhance the pairwise discriminative power of contrastive learning for sentence embeddings by combining PLMs with energy-based learning.
Outcome:	Empirical results show that the proposed method improves on seven standard semantic textual similarity tasks and a domain-shifted STS task.

Copied to clipboard

Wei Song, Ziyao Song, Ruiji Fu, Lizhen Liu, Miaomiao Cheng, Ting Liu

Are Neural Networks Extracting Linguistic Properties or Memorizing Training Data? An Observation with a Multilingual Probe for Predicting Tense (2021.eacl-main)

Challenge:	Despite its importance, discourse element identification is challenging due to the ambiguity of sentences . the number of elaboration sentences could be 10 times more than the number edna sentences.
Approach:	They propose to use sentence positional encodings to explicitly represent sentence positions and inter-sentence attentions to capture sentence interactions and enhance sentence representation.
Outcome:	The proposed model improves on a Chinese and English dataset.

Copied to clipboard

Bingzhi Li, Guillaume Wisniewski

Neural Extractive Summarization with Hierarchical Attentive Heterogeneous Graph Network (2020.emnlp-main)

Challenge:	a recent study has shown that neural networks can learn from linguistic representations without supervision . many studies have tried to identify which linguistic properties are encoded in the embeddings .
Approach:	They evaluate the ability of Bert embeddings to represent tense information . they use a multilingual linguistic probe to predict the morphology of a word .
Outcome:	The proposed model can predict tenses in French and Chinese, but the results drop sharply for Chinese.

Copied to clipboard

Ruipeng Jia, Yanan Cao, Hengzhu Tang, Fang Fang, Cong Cao, Shi Wang

Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance? (2021.eacl-main)

Challenge:	Existing extractive summarization methods focus on balancing salience and redundancy between sentences.
Approach:	They propose a hierarchical attentive heterogeneous graph for text summarization that models sentences . they propose to iteratively refine the sentence representations and deliver the labels by message passing .
Outcome:	The proposed method outperforms existing extractive summarization methods on large corpus.

Copied to clipboard

Abhilasha Ravichander, Yonatan Belinkov, Eduard Hovy

Textual Inference

Challenge:	Neural models have established state-of-the-art performance on several NLP benchmarks, but little is understood about the mechanisms by which they operate.
Approach:	They examine the probing paradigm through a set of controlled synthetic tasks and show that pretrained word embeddings play a considerable role in encoding these properties rather than the training task itself.
Outcome:	The proposed model can encode linguistic properties above chance-level even when distributed in the data as random noise, reversing the interpretation of absolute claims on probing tasks.

Conditional Semantic Textual Similarity via Conditional Contrastive Learning (2025.coling-main)

Copied to clipboard

Xinyue Liu, Zeyang Qin, Zeyu Wang, Wenxin Liang, Linlin Zong, Bo Xu

Aligning Cross-lingual Sentence Representations with Dual Momentum Contrast (2021.emnlp-main)

Challenge:	Existing methods to assess similarity between sentences encounter over-estimation problem . compared to fuzzy representations, similarity is comparatively lower in terms of "The person's age".
Approach:	They propose a conditional contrastive learning framework that constructs positive and negative samples from two perspectives.
Outcome:	The proposed method achieves state-of-the-art performance with five models based on bi-encoder and tri-encoding architectures.

Copied to clipboard

Liang Wang, Wei Zhao, Jingming Liu

Context-Aware Interaction Network for Question Matching (2021.emnlp-main)

Challenge:	Existing work uses sentences within the same batch as negatives, which suffers from easy negatives.
Approach:	They propose to align sentence representations from different languages into a unified embedding space . they adapt MoCo to further improve the quality of alignment .
Outcome:	The proposed model achieves state-of-the-art on several tasks.

Copied to clipboard

Zhe Hu, Zuohui Fu, Yu Yin, Gerard de Melo

Challenge:	Existing models focus on word-level local matching and neglect the importance of contextual information.
Approach:	They propose a context-aware interaction network to properly align two sequences and infer their semantic relationship by using gate fusion layers.
Outcome:	The proposed model can accurately align two sequences and infer their semantic relationship on two question matching datasets.

Efficient Nearest Neighbor Emotion Classification with BERT-whitening (2022.emnlp-main)

Copied to clipboard

Wenbiao Yin, Lin Shang

Information Retrieval

Challenge:	Existing methods for retrieval-based methods use pre-trained models for similarity search, but they perform poorly in retrieving semantically similar sentences.
Approach:	They propose a non-parametric emotion classification method using nearest neighbor retrieval that can be added to any pre-trained model without further training.
Outcome:	The proposed method improves the pre-trained model by 1.17 F1-macro on two emotion classification datasets.

Improving Contrastive Learning of Sentence Embeddings with Focal InfoNCE (2023.findings-emnlp)

Copied to clipboard

Pengyue Hou, Xingyu Li

Bipartite Graph Pre-training for Unsupervised Extractive Summarization with Graph Convolutional Auto-Encoders (2023.findings-emnlp)

Challenge:	SimCSE does not fully exploit the potential of hard negative samples in contrastive learning.
Approach:	They propose an unsupervised contrastive learning framework that combines SimCSE with hard negative mining to enhance the quality of sentence embeddings.
Outcome:	The proposed framework improves sentence embeddings on various STS benchmarks in terms of Spearman’s correlation, representation alignment and uniformity.

Copied to clipboard

Qianren Mao, Shaobo Zhao, Jiarui Li, Xiaolei Gu, Shizhu He, Bo Li, Jianxin Li

An Effective Label Noise Model for DNN Text Classification (N19-1)

Challenge:	Existing methods to rank sentences using pre-trained embeddings create a gap due to different optimization objectives.
Approach:	They propose a pre-trained embedding process that optimizes informative sentences . they use sentence-word bipartite graphs to model intra-sentential distinctive features .
Outcome:	The proposed model outperforms heavy BERT- or RoBERTa-based sentence ranking methods by providing summary-worthy representations.

Copied to clipboard

Ishan Jindal, Daniel Pressel, Brian Lester, Matthew Nokleby

Continual Learning for Sentence Representations Using Conceptors (N19-1)

Challenge:	Existing methods to train deep neural networks with label noise are limited to image classification models . label noise is important because of the large number of errors and errors in training datasets .
Approach:	They propose a non-linear processing layer that models label noise into a convolutional neural network (CNN) they add a noise model layer on top of their target model to account for label noise .
Outcome:	The proposed approach is robust to label noise and can learn better sentences . it is based on extensive experiments on text classification datasets .

Copied to clipboard

Tianlin Liu, Lyle Ungar, João Sedoc

A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space (2022.acl-long)

Challenge:	Existing sentence encoders for distributed representations of sentences are limited in their performance on fixed corpora.
Approach:	They propose a continual learning scenario for distributed representations of sentences . they initialize sentence encoders with corpus-independent features and update them sequentially .
Outcome:	The proposed sentence encoder can learn features from new corpora while maintaining its competence on previously encountered corporales.

Copied to clipboard

Yuhao Zhang, Hongji Zhu, Yongliang Wang, Nan Xu, Xiaobo Li, Binqiang Zhao

Ditto: A Simple and Efficient Approach to Improve Sentence Embeddings (2023.emnlp-main)

Challenge:	Existing methods for learning sentence representations focus on constitution of positive and negative representation pairs and do not focus on training objective.
Approach:	They propose a new method to learn sentence representations using BERT-like pre-trained models . they use a pairwise discriminating power and a model to model the entailment relation of triplet sentences .
Outcome:	The proposed method outperforms the previous state-of-the-art on diverse sentence related tasks.

Copied to clipboard

Qian Chen, Wen Wang, Qinglin Zhang, Siqi Zheng, Chong Deng, Hai Yu, Jiaqing Liu, Yukun Ma, Chong Zhang

Towards Structure-aware Paraphrase Identification with Phrase Alignment Using Sentence Encoders (2022.coling-1)

Challenge:	Prior studies diagnose the anisotropy problem in sentence embeddings from pre-trained language models without fine-tuning.
Approach:	They propose an unsupervised method that weights words with model-based importance estimations and computes the weighted average of word representations from pre-trained models as sentence embeddings.
Outcome:	Empirical evaluations show that the proposed method can alleviate the anisotropy problem and improve various pre-trained models on the STS benchmarks.

Copied to clipboard

Qiwei Peng, David Weir, Julie Weeds

Pretrained Language Models for Sequential Sentence Classification (D19-1)

Challenge:	Existing paraphrase identification datasets exhibit high correlation between positive pairs and the degree of their lexical overlap.
Approach:	They propose to combine sentence encoders with an alignment component by representing each sentence as a list of predicate-argument spans and decomposing the sentence-level meaning comparison into the alignment between their spans.
Outcome:	The proposed approach improves performance and interpretability for various sentence encoders.

Copied to clipboard

Arman Cohan, Iz Beltagy, Daniel King, Bhavana Dalvi, Dan Weld

Exploiting Invertible Decoders for Unsupervised Sentence Representation Learning (P19-1)

Challenge:	Recent successful models for document-level understanding have used hierarchical encoding and CRFs to capture dependencies between subsequent labels.
Approach:	They propose a pretrained language model that captures contextual dependencies without hierarchical encoding nor a CRF.
Outcome:	The proposed model captures contextual dependencies without hierarchical encoding nor a CRF on four datasets, including a new dataset of structured scientific abstracts.

Copied to clipboard

Shuai Tang, Virginia R. de Sa

Self-Attentive, Multi-Context One-Class Classification for Unsupervised Anomaly Detection on Text (P19-1)

Challenge:	Encoder-decoder models for unsupervised sentence representation learning discard decoder after training . decoded sentences are often used to make better predictions of words in a given sentence .
Approach:	They propose two types of decoding functions whose inverse can be easily derived without expensive inverse calculation.
Outcome:	The proposed models can learn good representations from encoders and decoders without expensive calculations.

Copied to clipboard

Lukas Ruff, Yury Zemlyanskiy, Robert Vandermeulen, Thomas Schnake, Marius Kloft

Debiased Contrastive Learning of Unsupervised Sentence Representations (2022.acl-long)

Challenge:	Existing methods for unsupervised anomaly detection use pre-trained word embeddings . proper text representation is critical for designing well-performing machine learning algorithms .
Approach:	They propose a new anomaly detection method that builds upon word embedding models to learn multiple sentence representations that capture multiple semantic contexts via the self-attention mechanism.
Outcome:	The proposed method performs on Reuters, 20 Newsgroups, and IMDB Movie Reviews datasets.

Copied to clipboard

Kun Zhou, Beichen Zhang, Xin Zhao, Ji-Rong Wen

Text Style Transfer with Contrastive Transfer Pattern Mining (2023.acl-long)

Challenge:	Recent studies have shown that contrastive learning improves pre-trained language models to derive high-quality sentence representations.
Approach:	They propose a framework to punish false negatives and generate noise-based negatives to guarantee the uniformity of the representation space.
Outcome:	The proposed framework improves pre-trained language models while pushing apart irrelevant negatives to guarantee the uniformity of the representation space.

Copied to clipboard

Jingxuan Han, Quan Wang, Licheng Zhang, Weidong Chen, Yan Song, Zhendong Mao

Text Generation

Challenge:	Existing methods for text style transfer only focus on the transformation between styles, yet they do not take into account that this transformation can be achieved via different hidden transfer patterns.
Approach:	They propose a novel approach which automatically mines hidden transfer patterns to improve TST . they use a clustering module to automatically discover hidden transfer pattern from the data .
Outcome:	The proposed method can be applied in a plug-and-play manner to enhance other methods to further improve their performance.

Exploring Anisotropy and Outliers in Multilingual Language Models for Cross-Lingual Semantic Sentence Similarity (2023.findings-acl)

Copied to clipboard

Katharina Haemmerl, Alina Fastowski, Jindřich Libovický, Alexander Fraser

Towards Debiasing Sentence Representations (2020.acl-main)

Challenge:	Recent studies have shown that contextual language models display outlier dimensions . this is true for monolingual and multilingual models, but little work has been done on multilingual contexts .
Approach:	They investigate outlier dimensions and their relationship to anisotropy in multilingual contexts . they focus on cross-lingual semantic similarity tasks .
Outcome:	The proposed model improves on cross-lingual semantic similarity tasks.

Copied to clipboard

Paul Pu Liang, Irene Mengze Li, Emily Zheng, Yao Chong Lim, Ruslan Salakhutdinov, Louis-Philippe Morency

Responsible & Trustworthy NLP

Challenge:	Recent work has shown word-level embeddings reflect and propagate social biases present in training corpora.
Approach:	They propose a method to debias word embeddings to reduce biases at sentence level . they hope their work will inspire future research on characterizing and removing biase .
Outcome:	The proposed method reduces biases and preserves performance on downstream tasks such as sentiment analysis and natural language understanding.

M3Seg: A Maximum-Minimum Mutual Information Paradigm for Unsupervised Topic Segmentation in ASR Transcripts (2023.emnlp-main)

Copied to clipboard

Ke Wang, Xiutian Zhao, Yanghui Li, Wei Peng

Text Generation

Challenge:	Topic segmentation aims to split automatic speech recognition transcriptions into segments that are bounded by thematic meanings.
Approach:	They propose a Maximum-Minimum Mutual information paradigm for linear topic segmentation without using any parallel data.
Outcome:	The proposed paradigm outperforms the state-of-the-art methods by a significant margin.

Reviews Meet Graphs: Enhancing User and Item Representations for Recommendation with Hierarchical Attentive Graph Neural Network (D19-1)

Copied to clipboard

Chuhan Wu, Fangzhao Wu, Tao Qi, Suyu Ge, Yongfeng Huang, Xing Xie

Pengwei Zhan, Jing Yang, He Wang, Chao Zheng, Xiao Huang, Liming Wang

Challenge:	Existing methods to learn user and item representations from review texts do not take into account the user-user and item-item relatedness of the user.
Approach:	They propose to use review content and user-item graphs to integrate them as different views.
Outcome:	The proposed approach can learn user and item representations from review content and user-item graphs.

Copied to clipboard

Universal Sentence Representation Learning with Conditional Masked Language Model (2021.emnlp-main)

Challenge:	Neural language models are vulnerable to word-level adversarial text attacks . previous word-based search methods assume important words influence prediction .
Approach:	They propose a method for similarizing the influence of words with contrast learning that encourages model to learn sentence representations in which words of varying importance have a more uniform influence on prediction.
Outcome:	The proposed method is compatible with various training methods and improves model robustness against various adversarial attacks.

Copied to clipboard

Ziyi Yang, Yinfei Yang, Daniel Cer, Jax Law, Eric Darve

Relating Simple Sentence Representations in Deep Neural Networks and the Brain (P19-1)

Challenge:	Existing methods to learn sentence representations on unlabeled corpora are difficult and expensive to obtain, making it hard to cover many domains and languages.
Approach:	They propose a method to train sentence representations on large unlabeled corpora by conditioning on the encoded vectors of adjacent sentences.
Outcome:	The proposed method outperforms existing models on SentEval and can be extended to a broad range of languages and domains.

Copied to clipboard

Sharmistha Jat, Hao Tang, Partha Talukdar, Tom Mitchell

Focus-Driven Contrastive Learning for Medical Question Summarization (2022.coling-1)

Challenge:	Existing deep learning models for natural language processing are not fully studied.
Approach:	They investigate whether deep recurrent models learn sentences against those encoded by the brain and whether there is any correspondence between hidden layers of these models and brain regions when processing sentences.
Outcome:	The proposed models can be used to synthesize brain data and improve subsequent stimuli decoding accuracy.

Copied to clipboard

Ming Zhang, Shuai Dou, Ziyang Wang, Yunfang Wu

Multi-Granularity Interaction Network for Extractive and Abstractive Multi-Document Summarization (2020.acl-main)

Challenge:	Existing methods to summarize health questions are not able to capture well question focus and lack the ability to understand sentence-level semantics.
Approach:	They propose a question focus-driven contrastive learning framework to capture question focus and exploit contrastive training at both encoder and decoder to obtain better sentence representations.
Outcome:	The proposed model achieves 5.33, 12.85 and 3.81 points over the baseline model on three medical benchmark datasets.

Copied to clipboard

Hanqi Jin, Tianming Wang, Xiaojun Wan

Less Mature is More Adaptable for Sentence-level Language Modeling (2025.acl-long)

Challenge:	Existing methods for document summarization use extractive and abstractive representations, but they don't take into account hierarchical structure of document clusters.
Approach:	They propose a multi-granularity interaction network for extractive and abstractive multi-document summarization which jointly learn semantic representations for words, sentences, and documents.
Outcome:	The proposed model outperforms baseline methods and achieves the best results on the Multi-News dataset.

Copied to clipboard

Abhilasha Sancheti, David Dale, Artyom Kozhevnikov, Maha Elbayad

Language Agnostic Multilingual Information Retrieval with Contrastive Learning (2023.findings-acl)

Challenge:	Existing studies fine-tune encoders or contrastive learning approaches to learn sentence representations.
Approach:	They propose to use sentence-level models to study how sentence representations influence downstream task performance.
Outcome:	The proposed models outperform token-level models in terms of time and data efficiency.

Copied to clipboard

Xiyang Hu, Xinchi Chen, Peng Qi, Deguang Kong, Kunlun Liu, William Yang Wang, Zhiheng Huang

Event Detection with Multi-Order Graph Convolution and Aggregated Attention (D19-1)

Challenge:	Annotated training data is costly to obtain in many languages .
Approach:	They propose a semantic contrastive loss to align parallel sentences that share the same semantics in different languages and a language contrastive gain to leverage parallel sentence pairs to remove language-specific information from non-parallel corpora.
Outcome:	The proposed model improves retrieval performance while requiring less computational effort.

Copied to clipboard

Haoran Yan, Xiaolong Jin, Xiangbin Meng, Jiafeng Guo, Xueqi Cheng

Syntactic Text Processing

Challenge:	Existing methods for event detection use first-order syntactic relations to identify trigger words.
Approach:	They propose a dependency tree-based method to model and aggregate multi-order syntactic representations in sentences.
Outcome:	The proposed method outperforms existing methods on a benchmark dataset . it uses a dependency tree based graph convolution network with aggregative attention .

Bilingual alignment transfers to multilingual alignment for unsupervised parallel text mining (2022.acl-long)

Copied to clipboard

Chih-chan Tien, Shane Steinert-Threlkeld

Multilinguality

Challenge:	a model trained to align only two languages can encode multilingually more aligned representations . a dual-pivot transfer theory is proposed for bilingual training .
Approach:	They propose methods for learning cross-lingual sentence representations using paired or unpaired bilingual texts.
Outcome:	The proposed models reach the state of the art in unsupervised bitext mining and perform better than multilingually supervised models.

Mere Contrastive Learning for Cross-Domain Sentiment Analysis (2022.coling-1)

Copied to clipboard

Yun Luo, Fang Guo, Zihan Liu, Yue Zhang

FaiMA: Feature-aware In-context Learning for Multi-domain Aspect-based Sentiment Analysis (2024.lrec-main)

Challenge:	Existing approaches to cross-domain sentiment analysis are labor-intensive and time-consuming.
Approach:	They propose a modified contrastive objective with in-batch negative samples to allow sentence representations from the same class to be pushed closer while those from the different classes become further apart in the latent space.
Outcome:	The proposed model can achieve state-of-the-art in cross-domain and multi-domain sentiment analysis tasks while transferring knowledge learned in the source domain to the target domain.

Copied to clipboard

Songhua Yang, Xinke Jiang, Hanjie Zhao, Wenxuan Zeng, Hongde Liu, Yuxiang Jia

Sentiment Analysis

Challenge:	Existing methods for aspect-based sentiment analysis are limited and integrating with existing techniques is difficult.
Approach:	They propose a framework that utilizes in-context learning as a feature-aware mechanism that facilitates adaptive learning in multi-domain ABSA tasks.
Outcome:	The proposed framework achieves significant performance improvements in multiple domains compared to baselines, increasing F1 by 2.07% on average.

CE-DA: Custom Embedding and Dynamic Aggregation for Zero-Shot Relation Extraction (2025.coling-main)

Copied to clipboard

Fu Zhang, He Liu, Zehan Li, Jingwei Cheng

Avoiding the Hypothesis-Only Bias in Natural Language Inference via Ensemble Adversarial Training (2020.emnlp-main)

Challenge:	Existing methods to predict relationships with given entity pairs are lacking in supervised methods.
Approach:	They propose a framework for zero-shot Relation Extraction that includes two modules: Custom Embedding and Dynamic Aggregation.
Outcome:	The proposed framework shows competitive performance on two ZSRE datasets.

Copied to clipboard

Joe Stacey, Pasquale Minervini, Haim Dubossarsky, Sebastian Riedel, Tim Rocktäschel

Responsible & Trustworthy NLP

Challenge:	Neural models pick up on annotation artefacts and spurious correlations, resulting in learning sentences that suffer from the same biases.
Approach:	They propose to tackle this problem by using adversarial training to reduce the bias in sentence representations by using an ensemble of adversaries.
Outcome:	The proposed approach produces more robust models outperforming previous de-biasing efforts when generalised to 12 other NLI datasets.

Unsupervised Domain Clusters in Pretrained Language Models (2020.acl-main)

Copied to clipboard

Roee Aharoni, Yoav Goldberg

Low-Resource NLP

Challenge:	Existing methods to select domains from large corpus of data are often over-simplistic and vague.
Approach:	They propose to use pre-trained language models to learn sentence representations that cluster by domains without supervision.
Outcome:	The proposed methods outperform established methods on domain selection and precision and recall with respect to an oracle selection.

HEGEL: Hypergraph Transformer for Long Document Summarization (2022.emnlp-main)

Copied to clipboard

Haopeng Zhang, Xiao Liu, Jiawei Zhang

Aspect-Category Enhanced Learning with a Neural Coherence Model for Implicit Sentiment Analysis (2023.findings-emnlp)

Challenge:	Abstract: Extractive summarization for long documents is challenging due to the extended structured input context.
Approach:	They propose a hypergraph neural network for extractive summarization by capturing cross-sentence relations.
Outcome:	The proposed model can capture cross-sentence relations and latent topics and keywords coreference, and section structure, and can be applied to scientific papers.

Copied to clipboard

Jin Cui, Fumiyo Fukumoto, Xinfeng Wang, Yoshimi Suzuki, Jiyi Li, Wanzeng Kong

Sentiment Analysis

Challenge:	Aspect-based sentiment analysis (ABSA) is a major research topic in NLP since social networking services have increased . but the recognition of implicit sentiments that do not contain obvious opinion words remains unexplored . elcom captures document-level coherence by using contrastive learning and sentence-level by a hypergraph .
Approach:	They propose aspect-category enhanced learning with a neural coherence model . it captures document-level coherency by contrastive learning and sentence-level by a hypergraph .
Outcome:	The proposed model captures document-level coherence by using contrastive learning and sentence-level by a hypergraph to mine opinions from explicit sentences to aid implicit sentiment classification.

Unsupervised Attention-based Sentence-Level Meta-Embeddings from Contextualised Language Models (2022.lrec-1)

Copied to clipboard

Keigo Takahashi, Danushka Bollegala

Representational Isomorphism and Alignment of Multilingual Large Language Models (2024.findings-emnlp)

Challenge:	Existing methods for creating metaembeddings from static word embeddings have been proposed, but they are not tied to a particular downstream task.
Approach:	They propose a sentence-level meta-embedding learning method that takes contextualised word embedding models and learns a phrase embeddable that preserves complementary strengths of the input source NLMs.
Outcome:	The proposed method outperforms existing methods on semantic textual similarity benchmarks on a supervised baseline and on token-level embeddings.

Copied to clipboard

Di Wu, Yibin Lei, Andrew Yates, Christof Monz

The Locality and Symmetry of Positional Encodings (2023.findings-emnlp)

Challenge:	Existing isomorphism of sentence representations can facilitate representational alignments in zero-shot and few-shot settings.
Approach:	They propose to apply a contrastive objective to LLMs with a small number of translation pairs to improve models' performance on Semantic Textual Similarity tasks.
Outcome:	The proposed representation-level approach significantly improves on Semantic Textual Similarity (STS) tasks across languages even without a monolingual objective.

Copied to clipboard

Lihu Chen, Gael Varoquaux, Fabian Suchanek

Language Models

Challenge:	Existing work uses fixed or learned positional encodings to infuse word order information into transformer-based language models.
Approach:	They conduct a systematic study of positional encodings in Bidirectional Masked Language Models.
Outcome:	The proposed model improves performance across a wide range of natural language understanding tasks.

BERT Has More to Offer: BERT Layers Combination Yields Better Sentence Embeddings (2023.findings-emnlp)

Copied to clipboard

MohammadSaleh Hosseini, Munawara Munia, Latifur Khan

Unsupervised Grouping of Public Procurement Similar Items: Which Text Representation Should I Use? (2024.lrec-main)

Challenge:	Obtaining sentence representations from BERT-based models is valuable as it takes less time to pre-compute a one-time representation of the data and then use it for the downstream tasks.
Approach:	They propose to combine certain layers of a BERT-based model rested on the data set and model to achieve substantially better results.
Outcome:	The proposed method outperforms baseline models on seven semantic textual similarity datasets and on eight transfer data sets.

Copied to clipboard

Pedro P. V. Brum, Mariana O. Silva, Gabriel P. Oliveira, Lucas G. L. Costa, Anisio Lacerda, Gisele Pappa

Low-Resource NLP

Challenge:	establishing reference prices is essential to guide competitors in setting product prices . however, selecting an appropriate representation for text is challenging .
Approach:	They propose a framework for text cleaning, extraction, and representation based on sentence representations for public procurement item descriptions.
Outcome:	The proposed approach captures the most important components of item descriptions.

3R: Enhancing Sentence Representation Learning via Redundant Representation Reduction (2025.emnlp-main)

Copied to clipboard

Longxuan Ma, Xiao Wu, Yuxin Huang, Shengxiang Gao, Zhengtao Yu

SemPA: Improving Sentence Embeddings of Large Language Models through Semantic Preference Alignment (2026.findings-acl)

Challenge:	Existing approaches to improve sentence representations lack fine-grained guidance on reducing redundant information.
Approach:	They propose a method that dynamically identifies redundant information from a dimensional perspective and trains the SRL model to redistribute semantics on different dimensions.
Outcome:	The proposed method improves sentence representations on seven semantic text similarity benchmarks.

Copied to clipboard

Ziyang Chen, Zhenxuan Huang, Yile Wang, Weiqin Wang, Lu Yin, Hui Huang